Skip to content

Implement Backups & Restore #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 9, 2025

Conversation

austin-denoble
Copy link
Contributor

@austin-denoble austin-denoble commented May 7, 2025

Problem

The Backups & Restore feature was added in the 2025-04 API, and needs to be implemented in the client.

Documentation: https://docs.pinecone.io/guides/manage-data/backups-overview

Solution

Backups and Restore functionality adds a number of new operations / methods under the Pinecone class.

Add new action functions and unit tests for the following:

  • createBackup
  • createIndexFromBackup
  • describeBackup
  • describeRestoreJob
  • listBackups
  • listRestoreJobs
  • deleteBackup

This PR also updates the README to include documentation around the feature, along with doc comments inline for all new methods, types, etc.

We skipped integration tests for this feature for the moment since it can take some time for a restoration job to complete. I've added detailed unit test coverage for basic piping between the interface and underlying OpenAPI internals, plus validation and error handling.

I've also done a fair bit of manual testing which I'll share below:

> await client.createBackup({ indexName: 'local-utility-index', name: 'my-backup', description: 'this is my backup' })
{
  backupId: '3df86526-0edf-41b0-a555-513effa2146d',
  sourceIndexName: 'local-utility-index',
  sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
  name: 'my-backup',
  description: 'this is my backup',
  status: 'Initializing',
  cloud: 'aws',
  region: 'us-east-1',
  dimension: undefined,
  metric: undefined,
  recordCount: undefined,
  namespaceCount: undefined,
  sizeBytes: undefined,
  tags: {},
  createdAt: '2025-05-08T14:52:00.236917882Z'
}

> await client.describeBackup('3df86526-0edf-41b0-a555-513effa2146d')
{
  backupId: '3df86526-0edf-41b0-a555-513effa2146d',
  sourceIndexName: 'local-utility-index',
  sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
  name: 'my-backup',
  description: 'this is my backup',
  status: 'Ready',
  cloud: 'aws',
  region: 'us-east-1',
  dimension: 5,
  metric: undefined,
  recordCount: 0,
  namespaceCount: 2,
  sizeBytes: 66877,
  tags: {},
  createdAt: '2025-05-08T14:52:00.236989Z'
}

> await client.listBackups()
{
  data: [
    {
      backupId: '3df86526-0edf-41b0-a555-513effa2146d',
      sourceIndexName: 'local-utility-index',
      sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
      name: 'my-backup',
      description: 'this is my backup',
      status: 'Ready',
      cloud: 'aws',
      region: 'us-east-1',
      dimension: 5,
      metric: undefined,
      recordCount: 0,
      namespaceCount: 2,
      sizeBytes: 66877,
      tags: {},
      createdAt: '2025-05-08T14:52:00.236989Z'
    },
    ...
  ],
  pagination: undefined
}

> await client.createIndexFromBackup({ backupId: '11450b9f-96e5-47e5-9186-03f346b1f385', name: 'my-restored-index' })
{
  restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b',
  indexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810'
}

> await client.listRestoreJobs()
{
  data: [
    {
      restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b',
      backupId: '11450b9f-96e5-47e5-9186-03f346b1f385',
      targetIndexName: 'my-restored-index',
      targetIndexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810',
      status: 'Pending',
      createdAt: 2025-05-08T14:55:12.749Z,
      completedAt: undefined,
      percentComplete: undefined
    },
    ...
  ],
  pagination: undefined
}

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • Infrastructure change (CI configs, etc)
  • Non-code change (docs, etc)
  • None of the above: (explain here)

Test Plan

CI - external app test, integration tests, unit tests

For manual testing, you can pull this branch down and run things locally with npm run repl. Set your PINECONE_API_KEY as an env variable, and then run the following:

From the top level of the pinecone-ts-client repo:

# this utility creates a new index and seeds some namespaces with data
npx tsx ./utils/generateAndSeedIndex.ts

npm run repl
await init()

await client.createBackup({ indexName: 'local-utility-index' })
await client.describeBackup('backup-id')
await client.listBackups()
await client.createIndexFromBackup({ backupId: 'backup-id' })
# etc

@austin-denoble austin-denoble changed the base branch from main to 2025-04 May 7, 2025 17:09
Copy link

@rohanshah18 rohanshah18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left a few comments on how you managed list backups for an index and list all backups for a project since its not clear to me.
Rest looks good, thanks for making this change!!

*/
export interface ListBackupsOptions {
/**
* The index name to list backups for. If not provided, all project backups will be listed.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't indexName a required field since its a path param?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. There are two different resources for listing backups: https://github.com/pinecone-io/apis/blob/888bea7d82dc40f9c9252b80fb1f6de5b3f36ed1/src/release/db/control/control.yaml#L42-L57

  • GET /indexes/{index_name}/backups - index level
  • GET /backups - project level

I just combined the two to prevent having two different methods, and instead allow the user to determine which they're targeting through the optionality of indexName. It calls the different operations under the hood: https://github.com/pinecone-io/pinecone-ts-client/pull/342/files#diff-82e31a389ec77273344d96a2e3f0ea165d1f6f8b7f0fbaf9fe55a42c489fbf5eR29-R32

It does seem like there's a problem with the spec (or possibly implementation on the server) here because limit and paginationToken are not defined as arguments to the listProjectBackups operation. We should follow up on that.

// }
```

`listBackups` lists all the backups for a specific index, or your entire project. If an `indexName` is provided, only

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as before, isn't indexName a required field since its a path parameter?

@austin-denoble austin-denoble merged commit 9d61268 into 2025-04 May 9, 2025
26 checks passed
@austin-denoble austin-denoble deleted the adenoble/implement-backups-restore branch May 9, 2025 04:04
@austin-denoble austin-denoble mentioned this pull request May 9, 2025
7 tasks
austin-denoble added a commit that referenced this pull request May 9, 2025
## Problem
The Backups & Restore feature was added in the `2025-04` API, and needs
to be implemented in the client.

Documentation:
https://docs.pinecone.io/guides/manage-data/backups-overview

## Solution
Backups and Restore functionality adds a number of new operations /
methods under the Pinecone class.

Add new action functions and unit tests for the following:
- `createBackup`
- `createIndexFromBackup`
- `describeBackup`
- `describeRestoreJob`
- `listBackups`
- `listRestoreJobs`
- `deleteBackup`

This PR also updates the `README` to include documentation around the
feature, along with doc comments inline for all new methods, types, etc.

We skipped integration tests for this feature for the moment since it
can take some time for a restoration job to complete. I've added
detailed unit test coverage for basic piping between the interface and
underlying OpenAPI internals, plus validation and error handling.

I've also done a fair bit of manual testing which I'll share below:

```
> await client.createBackup({ indexName: 'local-utility-index', name: 'my-backup', description: 'this is my backup' })
{
  backupId: '3df86526-0edf-41b0-a555-513effa2146d',
  sourceIndexName: 'local-utility-index',
  sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
  name: 'my-backup',
  description: 'this is my backup',
  status: 'Initializing',
  cloud: 'aws',
  region: 'us-east-1',
  dimension: undefined,
  metric: undefined,
  recordCount: undefined,
  namespaceCount: undefined,
  sizeBytes: undefined,
  tags: {},
  createdAt: '2025-05-08T14:52:00.236917882Z'
}

> await client.describeBackup('3df86526-0edf-41b0-a555-513effa2146d')
{
  backupId: '3df86526-0edf-41b0-a555-513effa2146d',
  sourceIndexName: 'local-utility-index',
  sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
  name: 'my-backup',
  description: 'this is my backup',
  status: 'Ready',
  cloud: 'aws',
  region: 'us-east-1',
  dimension: 5,
  metric: undefined,
  recordCount: 0,
  namespaceCount: 2,
  sizeBytes: 66877,
  tags: {},
  createdAt: '2025-05-08T14:52:00.236989Z'
}

> await client.listBackups()
{
  data: [
    {
      backupId: '3df86526-0edf-41b0-a555-513effa2146d',
      sourceIndexName: 'local-utility-index',
      sourceIndexId: '1d18673a-ed0e-4cdc-946f-541560367abf',
      name: 'my-backup',
      description: 'this is my backup',
      status: 'Ready',
      cloud: 'aws',
      region: 'us-east-1',
      dimension: 5,
      metric: undefined,
      recordCount: 0,
      namespaceCount: 2,
      sizeBytes: 66877,
      tags: {},
      createdAt: '2025-05-08T14:52:00.236989Z'
    },
    ...
  ],
  pagination: undefined
}

> await client.createIndexFromBackup({ backupId: '11450b9f-96e5-47e5-9186-03f346b1f385', name: 'my-restored-index' })
{
  restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b',
  indexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810'
}

> await client.listRestoreJobs()
{
  data: [
    {
      restoreJobId: '3de68445-77bf-4a74-9790-440523b5213b',
      backupId: '11450b9f-96e5-47e5-9186-03f346b1f385',
      targetIndexName: 'my-restored-index',
      targetIndexId: '8987dd9f-f3e4-49a3-bbd2-3d73cb9c2810',
      status: 'Pending',
      createdAt: 2025-05-08T14:55:12.749Z,
      completedAt: undefined,
      percentComplete: undefined
    },
    ...
  ],
  pagination: undefined
}
```


## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] This change requires a documentation update
- [ ] Infrastructure change (CI configs, etc)
- [ ] Non-code change (docs, etc)
- [ ] None of the above: (explain here)

## Test Plan

CI - external app test, integration tests, unit tests

For manual testing, you can pull this branch down and run things locally
with `npm run repl`. Set your `PINECONE_API_KEY` as an env variable, and
then run the following:

From the top level of the `pinecone-ts-client` repo:
```bash
# this utility creates a new index and seeds some namespaces with data
npx tsx ./utils/generateAndSeedIndex.ts

npm run repl
await init()

await client.createBackup({ indexName: 'local-utility-index' })
await client.describeBackup('backup-id')
await client.listBackups()
await client.createIndexFromBackup({ backupId: 'backup-id' })
# etc
```



---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
  - https://app.asana.com/0/0/1209571416750577
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants